Software testing is, in many cases of software system development, a major time-consuming part of the process. A recent survey has indicated that 40-70% of the software development effort is devoted to the business of software testing and error recovery. Improvement in this area can therefore contribute significantly to an improvement in time-to-market for new or revised products.
Known automated software testing systems, known as test tools, are essentially interfaces which make recordings of inputs applied and outputs produced for many manually performed combinations of system transactions or functions and data for which the system behaves as expected. As a later time, when a new version of the software system has been developed, these recordings can be used for regression testing. The aim of such testing is to develop confidence that changes incorporated in the new system have not unintentionally altered transactions or functions which were already present in the original version.
In regression testing, the test tool plays back the earlier recordings to simulate a user, automatically applying the same inputs to the system, and automatically comparing the system outputs to those originally obtained.
For each system function, the test tool generates a report showing whether the outputs remain the same. If the outputs do remain the same, the system passes the test. The aim of regression testing is to develop confidence that changes incorporated in the new version have not unintentionally altered functions which were already present in the original version.
Automated test tools of this type remove some of the work associated with manual testing. However the process can still be labour intensive:
The data base must be set up before each test. PA1 To make each recording, the system must be run and system functions must be actually performed. Usually the number of combinations of functions and data is large, so nose-to-screen time is very high. Much of this work is repetitive and therefore error prone. PA1 Reports from the test runs are bulky. Reconciling test results is complex and takes a long time. PA1 Even trivial changes to functionality or screen layout may force complete re-recording. It is to be noted that the test tool records inputs to the system quite blindly. Since the program has no knowledge of the business functions involved, or the structure of the input, it cannot ease the repetitive burden on the QA engineer. PA1 preparing a set of test scripts, each able, when input to the software system via a software testing interface program, to prompt performance of a transaction or function for which the software system is designed; PA1 preparing a test plan which invokes a sequence of said test scripts, and includes associated parameter inputs for the test scripts and an expected output of the function or transaction of each test script; PA1 inputting the test plan to the software system via said software testing interface program and running the test plan; and PA1 logging an indication of one or more resulting outputs of the software system compared to expected output(s). PA1 preparing a test plan which invokes a sequence of test scripts selected from a set of test scripts each able, when input to the software system via a software testing interface program, to prompt performance of a transaction or function for which the software system is designed, the test plan further including associated parameter inputs for the test scripts and an expected output of the function or transaction of each test script; PA1 inputting the test plan to the software system via said software testing interface program and running the test plan; and PA1 logging an indication of one or more resulting outputs of the software system compared to expected output(s). PA1 inputting a test plan to the software system via a software testing interface program and running the test plan; and PA1 logging an indication of one or more resulting outputs of the software system compared to expected output(s); PA1 wherein said test plan invokes a sequence of test scripts and includes associated parameter inputs for the test scripts, and an expected output of the function or transaction of each test script, which test scripts are selected from a set of test scripts each able, when input to the software system via a software testing interface program, to prompt performance of a transaction or function for which the software system is designed. PA1 first memory means to store a set of test scripts each able, when input to the software system via a software testing interface program, to prompt performance of a transaction or function for which the software system is designed; PA1 second memory means to store one or more test plans each invoking a sequence of said test scripts and including associated parameter inputs for the test scripts and an expected output of the transaction or function of each test script; PA1 computing means, including installed user interface software and an installed software testing interface program, and operatively connectable to said first and second memory means, for selectively inputting and running the test plan(s) to the software system via the software testing interface program; and PA1 logging means for logging an indication of one or more resulting outputs of the software system compared to expected output(s). PA1 inputting a test plan to the software system via a software testing interface program; and PA1 logging an indication of one or more resulting outputs of the software system compared to expected output(s); PA1 wherein said test plan invokes a sequence of test scripts selected from a set of test scripts each able, when input to the software system via a software testing interface program, to prompt performance of a transaction or function for which the software system is designed; and PA1 wherein said inputting of the test plan to the software system utilises user interface software which presents each test script to the software testing interface program so that the latter reads the script as a recording made earlier by the software testing interface program. PA1 first memory means to store a set of test scripts each able, when input to the software system via a software testing interface program, to prompt performance of a transaction or function for which the software system is designed; PA1 second memory means to store one or more test plans each invoking a sequence of said test scripts; PA1 computing means, including installed user interface software and an installed software testing interface program, and operatively connectable to said first and second memory means, for selectively inputting the test plan(s) to the software system via the software testing interface program and running the test plan; and PA1 logging means for logging an indication of one or more resulting outputs of the software system compared to expected output(s); PA1 wherein the user interface software is arranged for presenting each test script to the software testing interface program so that the latter reads the script as a recording made earlier by the software testing interface program.
Known software testing tools include Microsoft Visual Test, and SQA Robot (trade marks). These are respectively supplied by Microsoft Corporation, and SQA. Inc. of Woburn, Mass.
It is an object of the invention to provide an improved method of determining the functionality of a software system which at least in part alleviates the aforementioned disadvantages of methods utilising known test tool software.
The invention stems from an appreciation that the repetitive burden with conventional automated tools arises in part because the test tool software records inputs to the system under test quite blindly, and has no knowledge of the functions involved or the structure of the input. The method of the invention, in contrast, entails a practical strategy for building in knowledge of the functions of the software system under test. By applying this knowledge, the method is able to significantly reduce testing effort.
On one view, the invention replaces what were known as "scripts" in prior testing tools with a two-step process utilising test scripts which preferably focus on basic transactions or functions, and test plans that flexibly invoke a sequence of the test scripts. On another view, this approach allows the variable data to be separated from the test scripts, thus avoiding a source of inflexibility in the conventional approach.
In a first aspect, the invention provides a method of determining the functionality of a software system, including:
In this aspect of the invention, there is also provided a method of determining the functionality of a software system, including:
The invention still further provides, in its first aspect, a method of determining the functionality of a software system, including:
In a second aspect, the method includes the further step of generating a response indicative of functionality of a software system. The response may include generation of a log of representative outputs indicative of the functionality of the software system. The response may further alternatively include a change to the state of a database.
In a third aspect, the invention provides computer apparatus for determining the functionality of a software system, including:
Said inputting of the test plan to the software system preferably utilises user interface software which communicates with the software testing interface program as if it were a recording made earlier by the software testing interface program. The user interface software invokes the test script identified in the test plan and advantageously presents each script to the software testing interface program so that the latter also reads the script as a recording made earlier by the software testing interface program.
In a fourth aspect, the invention provides a method of determining the functionality of a software system, and/or of generating a response indicative of functionality, including:
In a fifth aspect, the invention provides a computer apparatus for determining the functionality of a software system, including:
Preferably, the parameter inputs for test scripts after the first include parameter inputs derived from recorded outputs of earlier test scripts. Also preferably, generator utility functions are employed in test plans to uniquely generate and/or identify parameter values. These techniques assist in minimising hardcoding of inputs to test scripts, and instead optimise parameter decoupling.
Preferably, the test plan(s) are written in a characteristic language utilising names for the respective test scripts, an established sequence for parameter inputs, identifiers for the respective parameters, parameter generator notation, and parameter broker notation instructing derivation of parameter inputs from recorded outputs of other test scripts.
Advantageously, operation of each of the test plans includes recordal of an inheritance file containing the values returned by test scripts while the test plan was running. An instruction, eg a special test script, may provide for access to the values in inheritance files. Later, inheritance files may include both their own values and the values inherited from an earlier inheritance file.
Test scripts are preferably written in the language of the software system under test, and may include a mix of general routines selected from the language, and routines specific to the system under test and often also to the transaction which the test script is intended to mirror.
The user interface software, in any of the aspects of the invention, preferably includes at least a graphical user interface, a test script definition language parser, a test plan parser which generates a set of action codes for each test plan, and an interpreter which responds to the action codes and interacts with the software system under test.
The software testing interface program is preferably a known program, eg SQA Suite incorporating SQA Robot and SQA Test Log Viewer, supplied by SQA, Inc of Woburn, Mass., or MS Test, supplied by Microsoft Corporation.
Preferably, different personnel effect the steps of preparing the test scripts, preparing the test plan(s), and operating the test plan(s).
It is preferred that the steps of preparing transaction and test plans be effected in parallel with the analysis/design and construction stages of creating the software system whose functionality is being determined.